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INTRODUCTION 


1.1  Objective  and  Scope 

The  objectives  of  this  effort  were  to  identify  RADC 
software  development  and  acquisition  needs,  to  investigate 
and  analyze  software  technology  methodologies  and  tools  to 
help  meet  these  needs  and  to  recommend  approaches  to  assist 
in  managing  the  acquisition  of  systems  where  software  is  an 
integral  part  of  the  system. 

Meetings  were  held  with  RADC  engineers  and  project 
managers  to  ascertain  the  types  of  software  problems 
encountered  during  the  acquisition  of  their  systems  and 
technical  and  management  methods  employed  during  this 
acquisition  process.  A  4  1/2-day  software  acquisition  and 
management  course  was  designed  based  upon  RADC  requirements. 
This  course  was  conducted  twice  during  this  effort  to  45 
RADC  engineers  and  computer  scientists  and  an  overview  of 
the  course  was  briefed  to  22  RADC  managers  and  procurement 
personnel.  An  evaluation  was  performed  to  determine  the 
effectiveness  of  this  approach  to  meet  the  RADC  acquisition 
needs . 

1 . 2  Recommendations 

The  following  recommendations  are  made  based  upon 
our  understanding  of  RADC  software  requirements  as  acquired 
through  designing  and  conducting  the  course  and  from  an 
evaluation  of  student  feedback: 

o  Offer  a  3  1/2-day  course  every  four  months.  For 
planning  purposes,  provide  prospective  students 
with  a  course  syllabus  at  least  one  month  before 
the  course  is  to  be  taught.  Include  in  this 
syllabus  a  description  of  the  prerequisite 
knowledge  required  of  the  student  and  provide  a 
list  of  recommended  readings. 

o  Refine  the  course  materials  to  include  more 
tutorial  type  information  in  the  beginning 
sessions . 
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o  Determine  the  feasibility  of  conducting  a  four 
hour  course  oriented  toward  inexperienced 
students.  Include  introductory  materials  on  the 
software  engineering  process  using  DoD-STD-2167A 
as  the  basis  for  a  software  acquisition  life  cycle 
methodology. 


1 . 3  Report  Contents 

This  report  records  the  results  of  the  effort 
expended  in  surveying  and  analyzing  the  RADC  software 
acquisition  needs  and  designing,  conducting,  and  evaluating 
the  software  acquisition  courses.  The  report  contains  four 
main  sections  including; 


Section  1 
Section  2 

Section  3 
Section  4 


This  introduction 

Results  of  meetings  with  government 
personnel 

Description  of  the  course 

Evaluation  of  the  course  by  the  students 
and  an  analysis  of  this  evaluation 
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2.  INFORMATION  GATHERING 


2 . 1  Approach 

Meetings  were  held  with  RADC  project  engineers, 
managers,  and  the  Directorate  Software  Focal  Points  to 
gather  information  on  the  types  of  software  intensive 
systems  currently  being  procured  at  RADC,  problems  that  are 
being  encountered,  and  management  and  technical  methods  used 
successfully  for  acquiring  this  software.  There  were  also 
meetings  at  the  Air  Force  Systems  Command  (AFSC)  Electronics 
System  Division  (ESD)  and  the  Mitre  Corporation  to  discuss 
management  and  technical  methodologies  and  tools  that  they 
have  used  in  the  procurements  at  ESD. 

Based  upon  the  results  of  these  meetings,  reviews  of 
the  current  DoD  Standards  and  practices,  the  experience  of 
the  consultants  from  this  effort,  and  surveys  of  software 
technology  literature,  an  approach  was  developed  for  a 
software  acquisition  course  to  be  used  as  a  training  vehicle 
for  RADC  engineers,  computer  scientists,  and  project 
managers.  This  approach  was  reviewed  with  RADC  personnel. 
Information  was  gathered  on  RADC  software  procurements  that 
could  be  used  as  examples  and  case  studies  for  the  software 
acquisition  course. 

Further  details  and  results  from  this  information 
gathering  task  are  included  in  this  section  of  the  report. 

2.2  RADC  Software  Activities 

Mr.  Richard  Motto,  RADC/COEE,  was  the  Technical 
Monitor  for  this  effort  and  provided  insight  into  the 
software  acquisition  process  at  RADC.  He  is  the  Mission 
Critical  Computer  Resource  (MCCR)  Software  Focal  Point  and 
represents  RADC  at  Air  Force  MCCR  software  meetings. 

Last  year,  Mr.  Motto  surveyed  the  five  mission 
directorates  to  characterize  their  software  acquisition 
activity.  Since  1985,  271  software  producing  efforts  have 
been  awarded  by  RADC,  broken  down  by  Directorate,  as 
follows : 
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131  Intelligence  &  Reconnaissance  (IR) 

7  6  Cor.mand  &  Control  (CO) 

25  Communications  (DC) 

23  Surveillance  (OC) 

16  Reliability  &  Compatability  (RB) 

Over  10  meetings  were  held  with  Mr.  Motto  over  the 
duration  of  the  contract  to  acquire  information  on  the 
software  acquisition  process  and  to  review  project  status. 
Meetings  were  held  with  the  mission  directorates  and 
Contractinq  (PK)  to  acquire  additional  information  on  their 
software  acquisition  problems  and  lessons  they  have  learned 
throughout  the  years.  Table  II-l  contains  a  list  of  these 
meetings  with  an  indication  of  purpose  of  the  meeting  and 
the  attendees. 

The  acquisition  needs  at  RADC  are  diverse  and  range 
from  acquiring  research  and  development,  proof -of-concept 
software  to  large  system  procurements  which  contain  both 
hardware  and  software  elements.  The  type  of  applications 
vary  extensively.  They  include,  but  are  not  limited  to, 
image  correlation,  database  systems,  signal  processing, 
smart  networks,  software  tools,  and  battle  management 
decision  aids.  Software  project  costs  range  from  $100,000 
to  $25  million.  Estimated  lines  of  code  developed  range 
from  10,000  to  800,000.  The  most  frequent  projects  are  in 
the  50,000  to  100,000  lines  of  code  range. 

Implementation  tools  used  for  assisting  in  the 
software  development  effort  include  standard  editors  and 
compilers,  performance  coverage  analyzers,  and  database 
management  systems.  Fortran,  Ada,  Pascal,  COBOL,  LISP,  C, 
and  assembly  are  the  common  programming  languages. 

Automated  project  management  tools,  test  tools,  and 
requirements\design  tools  are  used  on  the  larger 
procurements . 

Some  of  the  major  acquisition  problems  at  RADC 

include : 


o  The  lack  of  visibility  of  the  development  process 

o  The  difficulty  of  applying  DOD-STD-2167  and  DOD- 
STD-2167A  to  prototype  and  incremental  build 
developm.ents 
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TABLE  II -1.  RADC  MEETING  LIST 


DATE 

PLACE 

PURPOSE 

ATTENDEES* 

JULY 

18 

RADC/PK 

Acquisition  procurement 
issues 

A.Masercola, 

J.Marciniak 

19 

RADC/COEE 

Review  Draft  Outline 

3  DCT,  7  IR,  2  CO 
(see  list)** 

19 

RADC/ CO 

Report  to  Directorate 
on  status 

R.Urtz,  J.Marciniak, 
A. Vito 

22 

RADC/IRDW 

CATIS  fact  finding 

J. Frank,  P.Gwyther, 
A. Vito,  R. Motto, 

R. Hawkins 

AUGUST 

4 

RADC/ IRA 

3rd  Floor  S/w  problems 

ES  development 

J.Pletl,  etc. 

R. Motto,  A. Vito 

4 

RADC/IRRA 

1st  Floor  S/W  problems, 

ES  development 

Lt.Glen  Fye, 

R. Motto,  A. Vito 

4 

RADC/IRDW 

CATIS  documentation 
collection 

J. Frank,  P.Gwyther 

SEPTEMBER 

15 

RADC/ CO 

OC,  OP,  RB,  DC,  PK 
requirements 

F . Ahrens , T . Greci , 

K . Sherman , D . Motto , 

D . Lubecki 

NOTES 

*  L.  Duvall  attended  all  the  meetings 
**  July  19  attendees  -  M.Urbanik,  F.Rahrig,  G.Fye,  J. Weber, 


P. Langendorf ,  J. Frank,  R. Floyd,  J, Camera,  J.Pletl,  J.Marciniak,  A. Vito 
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o  The  need  for  contractual  mechanisms  for  changing 
requirements 

o  Contractors  do  not  follow  their  own  guidelines  and 
methodologies 

o  The  government  and  contractors  are  both  learning 
about  the  DOD  Standards  and  the  tailoring  process 

o  There  are  not  enough  resources  (dollars  and 
people)  to  do  the  proper  job,  and 

o  Engineers  are  inexperienced  and  need  to  be  kept 
abreast  on  software  technology  and  the  use  of  the 
DOD  standards 

Mr.  John  Frank,  IRDW,  shared  fteely  his  project 
anagement  experience  on  th<  acquisition  of  the  software  for 
he  Computer  Aided  Tactical  Intelligence  System  (CATIS) .  He 
riefed  us  on  the  lessons  learned  during  the  development 
phases  using  DOD-STD-2167  and  provided  extensive 
dccumentat ion  to  serve  as  examples  of  Data  Item  Descriptions 
and  project  management  r.ethods  used.  Figure  2-1  contains  a 
ii.st  of  the  documentation  supplied  categori'’ed  by  major 
acquisition  area.  Lessons  learned,  per  his  briefing, 
include : 

o  Requirements  must  be  uniquely  identified 
o  2167  Tailoring 

-  Govt  responsible  to  tailor  products  required 

-  Contractor  responsible  to  propose  DID  tailoring 
o  Establish  clear  program  organization 

o  Develop  "structured"  schedules 

o  Require  independent  QA,  CM  and  test  organizations 
o  Require  management  &  quality  metrics 
o  Review  contractor  understanding  of  DID's  via 
samples 

o  Require  structured  requirements  analysis 
o  Formalize  review  cycles 
o  Preview  review  packages 
o  Set  up  checklists  for  reviews 
o  Set  up  glossary  of  program  terms 
o  Document  control/word  processing  system  must 
support  2167  requirements 
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SYSTEM  ENGINEERING 

o  Review  Guidelines 

Software  Specification  Review  (SSR) 
Prolimina'^y  Design  Review  (PDP) 

Critical  Design  Review  (CDR) 
o  Software  Development  Plan  (SDP) 
o  Minutes  from  P:*"ogram  Status  Review 

REQUIREMENTS 

o  SSR  Guidelines 

o  Minutes  from  the  SSR  Technical  Review  meeting 
o  Requirements  Cross  Reference  Document 
o  Requiror lents  Tracing  Examples 
o  Structured  Analysis  Conformance  Memo 

DESIGN  -  Preliminary 

o  PDR  Guidelines 
o  Sample  Design  Document 
DESIGN  -  Detailed 

o  CDR  Guidelines 
o  Sample  Design  Document 
o  Data  Base  Design  Document 

LANGUAGE  ISSUES 

o  Software  Standards  and  Procedures  Manual 

TESTING 

o  Software  Test  Plan 
o  Example  Scenario  for  System  Test 

CONFIGURATION  MANAGEMENT 

o  Software  Configuration  Management  Plan 

QUALITY  ASSURANCE 

o  Software  Quality  Evaluation  Plan 

EVALUATION  AND  ASSESSMENT 

o  Management  Indicators 
o  ’  inutes  of  Program  Status  Review  #7 

SOFTWARE  MANAGEMENT 

o  Requests  for  Deviation/Waivers 
o  Program  Planning  Schedules 
o  Pert  Schedules 
o  Transition  Plan 

o  Risk  Analysis  -  Program  Status  Review  4 
o  Contract  Data  Requirements  List 

LESSONG  LEARNED 

o  J.  Fr~nk  briefing 

Figure  2-1  CATIS  DOCUMENTATION 
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2 . 3  ESD/MITRE  Experience 


2.3.1  ESD  Meeting 

Lorraine  Duvall  and  John  Marciniak  met  with  Mr.  Robert 
Kent  at  the  Electronics  Systems  Division  (ESD)  on  July  26,  1988 
to  discuss  ESD  software  acquisition  methodologies.  Mr.  Kent 
presented  a  version  of  a  briefing  on  the  DOD/ESD  software 
initiatives  which  included  discussion  revolving  around  the  DOD 
software  problem,  ESD  software  organizations,  and  the  main  ESD 
software  initiatives.  Below  is  a  summary  of  the  discussion  on 
three  of  these  areas  most  relevant  to  RADC  software  acquisition 
issues . 


Contractor  Capability  Evaluation 

ESD  sponsored  the  Contractor  Software  Engineering 
Asstissment  Task  v/ith  the  Software  Engineering  Institute  to 
leieiop  a  methodology  for  evaluating  the  capability  of  software 
engineering  contractors  before  contract  award.  A  second  purpose 
1  this  methodology  was  to  help  contractors  improve  their  process 
rcr  developing  software.  ESD  tested  this  evaluation  process  at 
Gunter  (Standard  System  Center)  and  for  Granite  Sentry  at 
SPAGECOM.  They  disc  used  this  for  the  STARS  Competing  Primes 
procurement . 

The  actual  assessments  were  done  through  the  ESD/MITRE 
Soft.-,are  Center.  The  SEI  offered  some  initial  guidance  on 
specific  methods.  They  found  that  this  audit  process  provided 
tnem  with  a  good  indication  of  the  contractors  capabilities  and 
should  be  considered  as  an  ongoing  process.  They  are  recommending 
a  central  facility  to  evaluate  technology  on  a  continual  basis. 

We  discussed  the  application  to  RADC  procurements.  This 
method  could  be  used  for  some  of  the  larger  RADC  efforts,  with 
assistance  provided  by  ESD.  Information  in  the  checklists  could 
be  used  by  RADC  engineers  to  formulate  their  evaluation  criteria 

MITRE/ESD  Software  Reporting  Metrics 

We  discussed  the  status  and  use  of  the  Mitre  developed 
software  m.dnaqement  metrics.  A  new  May  1988  report  (Software 
Management  Metrics,  H.  P.  Schultz,  May  1988)  includes  a  total  of 
11  mi-tric;-.  instead  of  the  original  eight.  This  report  also 
:ontains  additional  recommenua t i ons  for  reporting,  analysis, 
tailoring,  interpretation,  and  data  collection  and  has  been 
re'/io.f  d  to  be  (compatible  with  DCD-STD-2 167A . 
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The  software  metrics  have  been  used  extensively  on 
ESD  programs.  They  are  becoming  a  part  of  the  culture  and 
are  being  applied  because  they  are  useful.  Below  is  a  list 
which  summarizes  how  the  metrics  are  being  utilized: 

o  Historical  data  is  used  to  predict  future 
schedules  and  is  a  better  forecaster  than  the 
original  plans 

o  Unreasonableness  of  the  plans  can  be  illustrated 
graphically 

o  The  most  useful  metrics  are;  Size,  Personnel,  Development 
Progress,  and  Resource  Utilization 

o  The  least  useful  metrics  are:  Complexity  and 
Volatility 

The  data  items  are  placed  on  the  contractor  as  technical 

reports . 


Red  Team  Assessments 

Red  Team  assessments  have  been  conducted  on  approximately 
20  ESD  programs  since  1934.  These  programs  have  been  reviewed  and 
evaluted  to  determine  current  status  and  to  propose  problem 
resolutions.  The  methods,  questions  asked,  and  lessons  learned 
could  be  used  on  selected  RADC  programs,  as  was  done  on  CATIS. 

2.3.2  MITRE  Meeting 

Lorraine  Duvall  and  John  Marciniak  met  with  Judy  Clapp 
and  Richard  Sylvester  on  July  26,  1988  at  Mitre  to  discuss  the 
major  software  issues  at  ESD.  The  discussions  were  similar  to 
those  held  with  Robert  Kent,  with  some  additional  insight  from 
the  Mitre  perspective.  Below  is  a  summary  of  the  issues  raised 
at  the  meeting. 


A  MITRE  Software  Acquisition  Course 

We  obtained  a  copy  of  the  handouts  from  a  MITRE  produced 
course  entitled  "Software  Acquisition:  Policy,  Terminology,  and 
Standards".  This  course  provides  an  overview  of  the  government 
policies,  regulations,  and  standards  which  most  directly  affect 
the  software  aspects  of  ESD  acquisitions.  This  course  is  in  the 
process  of  being  revised  and  will  be  updated  to  include  2167a. 


9 


MITRE/ESD  Software  Reporting  Metrics 


The  four  new  metrics  (design  progress,  design  complexity, 
schedule  progress,  and  requirements  volatility)  are  beneficial 
because  they  report  more  items  up-front. 

Some  lessons  learned  include; 

o  There  has  been  improper  use  of  the  baselines,  i.e  the 
contractor  changes  the  plans  line  from  one  month  to  another. 

o  Contractor  people  on  the  management  side  do  not  take 
the  metrics  seriously. 

o  There  are  no  "tough"  metrics  -  you  have  to  cross 
reference  and  analyze  them  to  clearly  identify  the  problem  areas 
and  improvement  possibilities. 

o  They  are  sometimes  ignored  or  are  recorded  intermittently. 

o  Subcontractor  data  is  not  being  collected. 

o  The  reports  are  prepared  and  presented  by  the 
contractor  at  program  management  reviews. 

Contractors  are  beginning  to  use  Software  Control  Rooms 
tJiat  clearly  show  the  status  of  programs  (examples  are  Norden  in 
Norwalk  and  Grumman  in  Melbourne,  both  for  the  Joint  STARS 
Program) . 


DOD  Standards 

There  is  a  concern  about  the  amount  of  documentation 
required  and  the  amount  of  time  it  takes  the  government  to 
respond  to  contractor  submissions.  The  DIDs  need  to  be 
streamlinea  and  some  done  away  with. 

We  do  not  know  how  to  choose  the  right  number  of  CIs  or 
the  level  of  detail.  Their  experience  shows  that  when  design 
decisions  are  a  part  of  the  requirements,  there  may  be 
requirements  implied  that  need  to  be  explicitly  stated  rather 
than  buried  in  the  details  of  the  design. 

There  is  concern  about  tailoring  2167A.  They  are  waiting 
for  guidelines. 
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Miscellaneous  Items 


o  A  new  Air  Force  Study  Board  has  been  assembled  to  relook 
at  the  software  problem  from  a  1988  perspective. 

o  Mitre  is  focussing  on  a  two  step  competitive  process 
that  eliminates  a  basic  assumption  that  we  know  how  to  do  a  job 
before  doing  it. 

o  We  obtained  a  copy  of  the  February  issue  of  the 
ESD/MITRE  Software  Center  newsletter  which  presents  the  lessons 
learned  from  Graybeard  Source  Selection  visits. 
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3.  SOFTWARE  ACQUISITION  AND  MANAGEMENT  COURSE 


3 . 1  Course  Description 

A  course  was  designed  focusing  on  the  management  of  the 
software  engineering  process  in  the  RADC  environment.  The 
orientation  is  from  the  management  perspective,  using  DoD- 
STD-2167A  and  DoD-STD-2168  as  the  basis  for  a  software 
management  life  cycle  methodology. 

The  course  provided  an  overall  treatment  of  both  2167A 
and  2168  as  well  as  the  relevant  management  methodologies, 
e.g,,  configuration  management,  quality  assurance, 
measurement,  etc.,  that  comprise  the  environment.  Emphasis 
was  placed  on  the  contracting  process;  what  requirements 
should  government  place  in  the  statement  of  work,  how  should 
the  government  evaluate  the  industry  response,  and  what 
methods  are  best  utilized  throughout  the  development  process 
to  aid  in  communication  between  government  and  contractor 
personnel . 

Techniques  and  guidance  are  provided  for  the  statement 
of  work,  evaluation  of  the  corresponding  proposal,  and  the 
life  cycle  management  processes.  An  in-class  exercise  was 
utilized  to  build  confidence  in  the  management  process  and 
provide  examples  of  statement  of  work  elements  for  each 
major  topic  area.  Lessons  learned  from  previous  projects 
were  discussed  throughout  the  course.  Examples  from  the  IR 
program,  CATIS,  were  used  to  illustrate  the  application  of 
principles  introduced.  CATIS  documents  were  also  available 
for  review  during  the  course. 

TOPICS  AND  GENERAL  OUTLINE 

1.  DOD-STD-2 167A .  The  standard  is  described  using  an 
evolutionary  building  process  to  develop  the  entire  set  of 
methodologies  into  a  full  management  environment. 

Comparisons  are  made  between  2167A  and  its  predecessor, 

2167  . 

o  Understand  the  meaning  of  the  Software  Life 
Cycle  and  how  it  fits  into  the  Systems  Life 
Cycle 

o  Overall  understanding  of  2167A  and  the 
differences  from  2167 

o  Understanding  the  relationships  of  the  various 
standards  used  in  the  development  of  software 
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o  Understanding  the  documentation  requirements  of 
2167A 

o  Understanding  how  the  various  methodologies, 
e.g.,  Testing,  Configuration  Management, 

Quality  Assurance, etc.  integrate  into  2167A 

2.  Configuration  Management.  This  session  develops  the 
formal  definitions  of  CM,  describes  how  CM  fits  into  the 

2 167 A  life  cycle  environment,  and  provides  treatment  of  CM 
in  real  life  situations. 

o  Understand  the  role  of  Configuration  Management 
in  a  2167A  environment 
o  Understand  the  Life  Cycle  issues 
o  Understand  the  role  of  CM  tools 
o  Understanding  of  organizational  issues 
o  Assessing  CM  -  government  prerogatives 

3.  Requirements  and  Design.  These  sessions  provide  an 
overview  of  the  methodologies  and  tools  used  in  defining 
requirements  and  developing  designs  in  the  context  of  2167A. 

o  Develop  an  appreciation  for  the  importance  of 
requirements  analysis  and  design  to  the  2167A 
Software  Life  Cycle 

o  Understand  how  existing  methods  and  tools  can  be 
used  in  support  of  requirements  analysis  and 
design  activities 

4.  Quality  Assurance.  This  session  develops  an  understanding 
of  what  QA  is  and  details  the  use  of  2168  and  how  it  fits  in 

a  2167A  environment. 

o  Basic  understanding  of  QA  and  its  employment  in 
accordance  with  2167A 

o  Understand  the  contractor  QA  organization  and  its 
relationship  to  the  project  organization 
o  How  to  build  QA  into  the  SOW  and  evaluate  QA 
activities 

5.  Test  &  Integration.  This  session  details  the  process  of 
testing  in  the  context  of  2167A  and  describes  the  testing 
environment . 

o  Understand  the  role  of  Test  and  Integration  in  a 
2167A  environment 

o  Understand  Informal  and  Formal  Testing 
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o  Understand  the  importance  of  Test  Documentation 
o  Understand  how  existing  methods  and  tools  are  used 
in  support  of  Testing 

6.  Project  Management.  This  session  details  a  number  of 
management  issues  from  the  perspective  of  how  they 
apply  to  the  2167A  life  cycle  management  environment,  and 
provides  detailed  guidance  on  the  contracting  process  -  what 
should  the  government  contracting  agency  put  into  the 
statement  of  work  and  how  should  they  evaluate  the 
contractor  response.  Insights  are  provided  from  actual 
government  and  industry  programs. 

o  Understand  the  different  perspectives  of  the 
Contracting  Agency  and  the  Contractor 
o  Understanding  of  requirements  for  management 
processes  in  the  Statement  of  Work  and  Proposal 
o  Understanding  Data  Rights  Issues 
o  Understand  organizational  issues  and  how  they 
should  be  treated  in  the  Statement  of  Work 
o  Understand  Subcontractor  Management 
o  Techniques  for  Risk  Management 

7.  Evaluation  and  Assessment.  This  session  provides 
insight  into  how  the  government  seeks  to  provide  visibility 
into  project  status,  techniques  used,  etc.  Guidance  is 
provided  on  procedures  for  conducting  and  providing 
visibility  throughout  the  life  cycle  process. 

o  Formal  Reviews  and  Audits  -  Understanding  detailed 
procedures 

o  Understanding  the  State-of-the-Art  and  when,  if 
and  how  to  apply  measurements  throughout  the  life 
cycle 

o  Understand  how  to  apply  Software  Management  and 
Quality  Indicators 

o  Understand  what  IV&V  is  and  how  it  relates  to 
software  development 

8.  Tailoring  2167A.  This  session  deals  with  the  various 
aspects  of  tailoring  the  standard  from  the  government  and 
industry  perspectives.  It  covers  when,  why,  what  and  how  to 
tailor  including  phase  in  the  life  cycle,  size  of  the 
project,  and  funding  source. 
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3.2  Class  Sessions 

Two  classes  were  taught  to  a  total  of  45  RADC 
engineers  and  computer  scientists.  An  additional  two 
briefings,  providing  an  overview  of  the  course,  were  given 
to  22  RADC  managers  and  procurement  personnel. 

The  first  class  was  attended  by  27  personnel  from 
the  IR  Directorate,  one  from  the  Photonics  Laboratory,  and 
one  from  the  CO  Directorate.  Based  upon  the  feedback  from 
the  attendees  of  this  class  (see  Section  4  of  this  report 
for  a  summary  of  the  course  evaluations)  and  the  experience 
gained,  the  course  was  revised  and  given  to  17  personnel 
from  the  remaining  RADC  Mission  Directorates.  Eight  attended 
from  the  CO  Directorate,  and  three  each  from  OC,  RB,  and  DC. 

Each  class  was  held  for  4  1/2  days  with 
approximately  the  last  two  hours  of  three  of  the  days 
dedicated  to  working  on  the  class  exercise.  The  original 
schedule,  for  Class  1,  is  shown  in  Figure  3-1.  The  revised 
schedule  for  Class  2  is  contained  in  Figure  3-2. 

The  instructors  for  each  session  are  indicated  in 
parentheses  on  the  class  schedules.  Lorraine  Duvall,  John 
Marciniak,  and  Stuart  Hirshfield  were  instructors  for  both 
classes.  Two  additional  instructors  were  used  for  the  first 
class  for  IR  to  provide  special  insight  into  the  problems 
and  lessons  learned  from  IR  programs.  Armand  Vito,  a 
recently  retired  RADC  engineer,  discussed  the  problems  faced 
and  lessons  learned  in  recent  IR  procurements;  John  Frank 
from  IRDW  provided  invaluable  insight  from  his  experience  in 
managing  the  software  acquisition  of  CATIS. 

In  the  second  class,  configuration  management  was 
introduced  earlier  in  the  schedule  to  help  alleviate  some  of 
the  confusion  in  the  first  class.  Also,  introductory 
project  management  issues  were  spread  throughout  the  course 
rather  than  dedicating  an  earlier  session  to  them.  All  the 
management  measures  were  put  into  one  session,  and  the  DOD 
standards  sessions  were  reorganized. 

Course  materials  were  distributed  to  each  student 
and  consisted  of  approximately  500  pages  of  copies  of  the 
instructors'  viewgraphs,  DoD-STD-2 167A  and  2168  and  the 
associated  DIDs,  and  the  AFSC  Pamphlets  for  the  Software 
Quality  and  Management  Indicators. 
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MONDAY 


0800  -  0845 
0845  -  1000 


1000  -  1100 
1100  -  1200 

1200  -  1300 

1300  -  1430 
1430  -  1630 


TUESDAY 


0800  -  0900 
0900  -  0930 
0930  -  1100 


1100  -  1130 

1130  -  1230 

1230  -  1400 
1400  -  1630 


WEDNESDAY 


0800  -0930 


RADC  Software  Acquisition  and  Managenent 

Class  1 


SESSION  I  -  Introduction 

Course  Introduction  (LD) 

Lessons  Learned  (JF) 

SESSION  II  -  Systems  Engineering 

Systems  Acquisition/Software  Lifecycles  (JM) 
Introduction  to  the  Class  Exercise 

Lunch 

DOD-STD-2167A  (LD) 

Project  Management  (JM) 


SESSION  III  -  Requirements  Analysis 
Language  Issues  (SH) 

DOD-STD-2167A  Activities/Products  (LD) 
Requirements  Methodologies,  Tools  and 
Techniques  (SH) 

SESSION  IV  -  Design 

DOD-STD-2167A  Activities/Products  (LD) 

Lunch 

Design  Methodologies,  Tools  and  Techniques  (SH) 
Class  Exercise 


SESSION  V  -  Test  and  Integration 
Test  Methodologies,  Tools  and  Techniques  (JM) 


FIGURE  3-1 


CLASS  I  SCHEDULE 

16 


0930 


1130 


SESSION  VI  -  Configuration  Management 
CM  Methodologies,  Tools  and  Techniques  (JM) 
Lunch 


1130  -  1230 


SESSION  VII  ~  Quality  Assurance 


1230  - 

1400 

QA  Methodologies,  Tools  and  Techniques 

1400  - 

1430 

Software  Quality  Indicators  (LD) 

1430  - 

1630 

Class  Exercise 

THURSDAY 

0800  - 

0930 

SESSION  VIII  -  Project  Management 

Software  Management  Techniques  (JM) 

0930  - 

1000 

Management  Measures  (LD) 

1000  - 

1100 

Tailoring  2167A  (AV) 

1100  - 

1200 

Software  Risk  Management  (LD) 

1200  - 

1300 

Lunch 

1300  - 

1630 

Class  Exercise 

FRIDAY 

0800  - 

1000 

SESSION  IX  -  Review  and  Synthesis 

Class  Exercise  Review 

1000  - 

1100 

Lessons  Learned  (AV) 

1100  - 

1200 

Course  Summary  (LD) 

FIGURE  3-1  (continued)  CLASS  1  SCHEDULE 

1 


(JM) 
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RADC  Software  Acquisition  and  Nanagenent 

Class  2 


DAY/TIME 

SESSION 

TOPIC 

MONDAY 

0800  -  0900 

I 

Course  Introduction  (LD) 

0900  -  1000 

II-l 

Systems  Acquisition/Software  Lifecycles 

(JM) 

1000  -  1100 

II-2 

DOD  Standards  (LD) 

1100  -  1200 
1200  -  1300 

VI 

Configuration  Management  -  1  (JM) 

Lunch 

1300  -  1400 
1400  -  1630 

VI 

Configuration  Management  -  2  (JM) 
Introduction  to  Class  Exercise 

TUESDAY 

0800  -  0845 

III- l 

IV- 1 

2167A  Requirements/ Design  Activities 
and  Products  (LD) 

0845  -  1015 

II-4 

Language  Issues  (SH) 

1015  -  1145 

1145  -  1300 

III-2 

Requirements  Methodologies,  Tools  and 
Techniques  (SH) 

Lunch 

1300  -  1430 
1430  -  1630 

IV-2 

Design  Methodologies,  Tools  and  Techniques  (SH) 
Class  Exercise 

ffEIKIESDAY 

0800  -  0930 

V 

Test  Methodologies,  Tools  and  Techniques 

(JM) 

0930  -  1130 
1130  -  1300 

VII-1 

QA  Methodologies,  Tools  and  Techniques 
Lunch 

(JM) 

1300  -  1400 
1400  -  1630 

VII-2 

Software  Risk  Management  (LD) 

Class  Exercise 

THURSDAY 

0800  -  1000 

VIII-1 

Project  Management  (JM) 

1000  -  1100 

VIII-2 

Management  Measures  (LD) 

1100  -  1200 
1200  -  1300 

VIII-3 

Tailoring  2167A  (JM) 

Lunch 

1300  -  1430 
1430  -  1630 

VIII-4 

Evaluation  and  Assessment  (JM) 

Class  Exercise 

FRIDAY 

Class  Exercise  Review  (JM) 
Lessons  Learned  (LD) 


FIGURE  3-2  CLASS  2  SCHEDULE 
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0800 

1000 


1000 

1100  IX 


4. 


CX}URSE  EVALUATIONS 


4 . 1  Introduction 

On  the  last  day  of  each  course,  an  evaluation  form 
was  distributed  to  each  student  (Figure  4-1) .  There  were  ten 
questions  on  this  form.  The  first  four  questions  asked  the 
evaluator  to  rate,  on  a  five  point  scale,  the  course  in 
terms  of  job  benefits,  course  organization,  instructor 
effectiveness,  and  depth  of  material.  Questions  5  and  6 
asked  which  sessions  were  most  and  least  beneficial. 
Questions  8  and  9  were  directed  towards  obtaining  feedback 
on  the  best  and  worst  things  about  the  course;  question  7 
asked  for  recommendations  for  additional  material  that  the 
engineer  would  like  to  have  seen  covered;  and  question  10 
was  open  for  general  comment. 

The  number  of  forms  returned  per  class  are  as 

follows : 


Class  I  16  completed  forms 

Class  II  11  completed  forms 

This  section  of  the  report  summarizes  the  feedback 
from  this  evaluation  process  and  provides  an  analysis  based 
upon  the  variations  between  classes. 

4.2  Evaluation  Results 

Feedback  from  the  first  class  was  used  to  revise  the 
2nd  class,  which  is  reflected  in  the  evaluation  from  the 
students.  The  second  class  was  reorganized  to  provide  the 
introduction  of  basic  subject  matter  earlier  in  the  course, 
depth  was  added  to  others,  and  some  redundant  materials  were 
el iminated , 


OVERALL  RATINGS 

Tables  IV-1  and  IV-2  summarize  the  results  of  the 
first  four  questions  which  rated  aspects  of  the  course  on  a 
five  point  scale  from  Poor  to  Very  Good.  These  results 
indicate  that  the  second  class  was  more  satisfied  than  the 
first.  Fifty-six  percent  of  the  first  class  rated  the 
course  Good  or  Very  Good  (Questions  4  and  5)  in  terms  of 
being  beneficial  to  their  job  versus  81  percent  for  the 
second  class. 
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RADC  SOFTWARE  AND  ACQUISITION  AND  MANAGEMENT 
EVALUATION  QUESTIONAIRE 

1  2  3  4  5 

Poor  Not  So  Good  Neutral  Good  Very  Good 

Please  circle  the  appropriate  number. 

1.  How  benefical  will  this  course  be  to  your  job? 

1  2  3  4  5 

2.  How  was  the  course  organization? 

1  2  3  4  5 

3.  How  would  you  rate  instructor  effectiveness? 

1  2  3  4  5 

4.  How  would  you  rate  the  depth  of  the  material  in  meeting  your  needs? 

1  2  3  4  5 

5.  Which  sessions  were  of  most  benefit  to  you? 

6.  Which  sessions  were  of  least  benefit? 

7.  What  additional  material  would  you  like  to  have  seen  covered? 

8.  What  was  the  best  thing  about  the  course? 

9.  What  was  the  worst  thing  about  the  course  (and  how  would  you  change  it)? 

10.  Is  there  anything  else  you  would  like  to  tell  us?  (Please  continue 

on  the  back  if  you  need  to.) 

Figure  4-1.  Evaluation  Form 
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Table  TV-1 


COURSE  RATING  -  CLASS  1 


QUESTION 

RATING 

-  Percent 

Answering 

1 

2 

3 

4 

5 

l.Job  benefit 

12 

25 

50 

6 

2 . Organization 

12 

44 

38 

6 

3 . Instructors 

6 

44 

44 

4. Material  Depth 

19 

25 

50 

6 

Average 
(Questions  2 

% 

-  4) 

0 

12 

38 

44 

4 

Table 

IV-2 

COURSE  RATING  -  CLASS  2 

QUESTION 

RATING 

-  Percent 

Answering 

1 

2 

3 

4 

5 

l.Job  benefit 

18 

36 

45 

2 . Organization 

10 

36 

45 

10 

3  .  Instructors 

18 

6t 

18 

4. Material  Depth 

18 

10 

55 

10 

Average 

% 

0 

9 

21 

55 

13 

(Questions  2-4) 


2: 


A  better  evaluation  was  also  given  for  course 
organization,  instructor  effectiveness,  and  depth  of 
material  (Questions  2,  3,  and  4)  where  the  ratings  were  also 
significantly  higher.  The  biggest  differential  occurred  when 
evaluating  instructor  effectiveness.  Eighty-two  percent 
from  the  second  class  indicated  Good  or  Very  Good,  compared 
to  half  of  that,  or  44  percent,  in  the  first  class. 

The  average  percentage  for  Questions  2-4  are 
illustrated  in  the  bar  charts  in  Figures  4-2  and  4-3. 


INDIVIDUAL  SESSIONS 

Questions  5  and  6  asked  which  sessions  where  of  most 
and  least  benefit  to  the  student.  The  results  are  summarized 
in  Table  IV-3  by  question,  class,  and  session. 

Sessions  III  and  VIII  were  most  liked,  although  not 
significantly.  Answers  to  subsequent  sessions  provided  more 
insight  into  session  preference.  The  Exercise,  Tailoring, 
Management  Measures,  and  Lessons  Learned  were  specifically 
mentioned  as  the  best  things  about  the  course  (Question  8) . 
However,  in  Class  1,  it  was  stated  that  the  exercise  should 
have  been  more  applicable  to  IR.  A  request  was  also  made 
that  there  be  more  than  one  exercise,  taking  into  account 
the  different  project  sizes  and  the  varying  experience  of 
the  students. 


ADDITIONAL  MATERIAL 

Question  7  asked  what  additional  material  they  would 
like  to  have  seen  covered.  Topics  mentioned  for  class  1 
were : 

o  Guidelines  and  recommendations  for  standards  use 
o  Examples  of  principals  of  each  block 
o  Real  life  experiences,  lessons  learned 
o  Define  terms  and  concepts  up-front 
o  PK  relationships 

o  Specific  examples  related  IR  type  programs 

Many  of  the  topics  were  similar  from  class  2  and 
included : 


o  What  to  do  when  problems  arise 
o  The  "how”  of  monitoring  and  control 
o  List  of  acronyms  at  end  of  each  session 
o  Specific  tailoring  guidance 
o  Example  documents 
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40 

%  • 

30 

%• 

20 

%  ■ 

10 

% 

0 

% 

1  2  3  4  5 

Poor  Not  So  Good  Neutral  Good  Very  Good 

Figure  4-2  EVAUUATION  RATING  RESULTS  -  CLASS  1 

Organization,  Instructors,  Depth 


Poor  Not  So  Good  Neutral  Good  Very  Good 

Figure  4-3  EVALUATION  RATING  RESULTS  -CLASS  2 
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Table  IV-3  EVALUATION  RESULTS  -  SESSION  NOST/LEAST  BENEFICIAL 


MOST 

LEAST 

SESSION 

TOPIC 

Cl 

C2 

Cl 

C2 

2 

2 

2 

2 

All/Majority  (Most)  None  (Least) 

I 

Introduction 

1 

II-l 

Systems  Engineering 

1 

3 

1 

II-2 

DOD  Standards 

1 

1 

1 

1 

II-4 

Language  Issues 

4 

1 

III-l 

2167A  Requirements  Activities/Products 

3 

4 

1 

1 

III-2 

Requirements  Analysis 

2 

1 

IV-1 

2167A  Design  Activities/Products 

1 

1 

1 

1 

IV-2 

Software  Design 

V 

Test  and  Integration 

1 

2 

1 

VI 

Configuration  Management 

1 

2 

VII-1 

Quality  Assurance 

VII-2 

Software  Risk  Management 

2 

2 

1 

VIII-1 

Project  Management 

3 

2 

VIII-2 

Management  Measures 

4 

2 

2 

VIII-3 

DOD-STD-2167A  Tailoring 

2 

VIII-4 

Evaluation  and  Assessment 

1 

IX 

Software  Development  Lessons  Learned 

4 

1 

1 

Class  Exercise 

I 


NOTE:  Cl 


Class  1,  C2 


Class  2 
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o  More  time  on  2167A 

o  Define  reliability,  support,  tools,  etc.  up  front 
o  Include  SOW  skeletons  with  key  insertion 
checklists 

o  Course  material  complete 


BEST  AND  WORST  THINGS  ABOUT  THE  COURSE 


In  addition  to  the  sessions  mentioned  above  as  being 
one  of  the  best  things  of  the  course,  the  first  class  listed 
the  following,  as  an  answer  to  Question  8: 

o  Field  experience  and  class  discussions 
o  Knowledge  2167A;  terminology,  standards 
o  Totality  of  subject  covered  -  difficult  topic 
handled  well 

Class  2  did  not  list  any  individual  sessions  but 
made  the  following  comments  on  the  best  things  about  the 
course: 

o  Exposure  to  the  SW  procurement  process  and  open 
discussion  with  people  of  like  interests 
o  Exposure  to  tools/techniques  in  use  today 
o  Good  feel  for  2167A  s/w  development  environment 
o  Dynamic  lecturer 
o  Overall  structure,  very  clear 
o  What  I  learned,  everything 

o  Instructors  with  actual  background  in  government 
software  management 
o  Very  comprehensive 

o  Small  class  size,  vivid  discussion  based  on 
experience 

Question  9  asked  what  was  the  worst  thing  about  the 
course.  Answers  that  have  not  previously  been  indicated 
under  the  session  discussion  include,  from  class  1: 

o  Some  lectures  too  indepth  -  not  general  enough 
o  Quality  of  instruction,  boring,  too  much 
philosophy 

o  Course  topic  of  government  standards 
o  Actual  application  of  standards 
o  Mix  of  student  experience  detracted 
o  Bad  SOW  advice 

o  Not  having  been  part  of  a  $20  -  30  million  program 
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Class  2  suggested  the  following  on  organization: 


o  Needs  overall  thread  -  topics  jumped  around 
o  Lack  of  a  good  introduction  aimed  at  non-software 
personnel 
o  Too  long 

o  No  allowance  for  doing  regular  job  during  the  week 
of  the  course 

One  respondee  from  Class  2  complained  that  the 
course  addressed  large  scale  procurements  and  not 
procurements  of  the  scale  of  RADC.  Another  wanted  more 
material  on  how  to  apply  and  tailor  the  standard. 


OTHER  SUGGESTIONS 


Additional  suggestions  from  class  1,  not  already 
mentioned  in  relationship  to  other  course  recommendations, 
were  directed  to  having  two  courses  to  handle  the  diversity 
of  experience  of  the  students  where  smaller  efforts  could  be 
discussed  specific  to  their  needs.  There  was  also  a 
suggestion  that  there  be  fewer  briefings,  more  application 
exercises,  and  more  experienced  instructors  who  have  applied 
the  theory. 

Some  of  the  additional  suggestions  from  class  2 
reflected  the  mix  of  hardware  and  software  backgrounds.  One 
respondee  suggested  that  the  course  be  spit  into  two  -  one 
for  hardware  and  one  for  software  personnel  and  that  each 
course  be  a  total  of  three  days.  Another  said  the 
assumptions  should  not  be  made  that  people  have  software 
background.  Other  recommendations  from  class  2  included: 

o  Compare  how  procurement  is  being  done  to  how  it 
should  be  done 

o  A  condensed  version  or  general  overview  should  be 
presented  to  all  engineering  and  computer  science 
personnel  at  RADC 

o  Suggest  training  be  a  condition  of  civilian 
promotion  (like  in  the  military) 
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MISSION 

of 

Rome  Air  Development  Center 


R/iDC  plans  and  executes  research,  development,  test  and  selected 
acquisition  programs  in  support  of  Command,  Control,  Communications 
and  Intelligence  (C^I)  activities.  Technical  and  engineering  support  within 
areas  of  competence  is  provided  to  ESD  Program  Offices  (FOs)  and  other 
ESD  elements  to  perform  effective  acquisition  of  CT  systems.  The  areas 
of  technical  competence  include  communications,  command  and  control, 
battle  management,  information  processing,  surveillance  sensors, 
intelligence  data  collection  and  handling,  solid  state  sciences, 
electromagnetics,  and  propagation,  and  electronic,  maintainability,  and 
compatibility. 


